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Abstract 

We consider the deformation of a discontinuous group acting on 
the Euclidean space by affine transformations. A distinguished feature 
here is that even a 'small' deformation of a discrete subgroup may 
destroy proper discontinuity of its action. In order to understand 
the local structure of the deformation space of discontinuous groups, 
we introduce the concepts from a group theoretic perspective, and 
focus on 'stability' and 'local rigidity' of discontinuous groups. As 
a test case, we give an explicit description of the deformation space 
of Ij k acting properly discontinuously on by affine nilpotent 

transformations. Our method uses an idea of 'continuous analogue' 
and relies on the criterion of proper actions on nilmanifolds. 
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1 Local rigidity and stability 



Our concern in this article is with the deformation of discontinuous groups 
for non-Riemannian homogeneous spaces. 

1.1 Deformation of discontinuous groups 
— the non-Riemannian case 

In contrast to the traditional case of discontinuous groups acting on Rie- 
mannian manifolds as isometries, our problem in the non-Riemannian case 
includes the following subproblem: if a discrete subgroup can be deformed, 
determine the range of the deformation parameters that does not destroy the 
proper discontinuity of its action. 

As a clue to understanding the local structure of the 'deformation space', 
we consider a manifold X acted on by a Lie group G. Suppose T is a discon- 
tinuous group for X, that is, T is a discrete subgroup of G acting properly 
discontinuously and freely on X. Let T' be another discrete subgroup of G 
which is 'sufficiently close' to T. Now, our basic question is if the following 
statements hold or not: 

(R)' : (Local Rigidity) V is conjugate to T by an inner automorphism of G. 

(S)' : (Stability) The T'-action on X is properly discontinuous and free. 

For a homogeneous space X = G/H (H being a closed subgroup of G), 
obvious remarks are: 

1) If H is compact, then (S)' automatically holds. 

2) If (R)' holds, so does (S)'. 

This article studies the deformation of discontinuous groups for G/H in 
the case that (S)' does not hold. This implies particularly that H is non- 
compact and that (R)' does not hold. 

Let us now formalize the above two statements (R)' and (S)' more rigor- 
ously. We begin with an (abstract) finitely generated group T, and denote 
by Hom(r, G) the set of all group homomorphisms of T into a Lie group G. 
Taking generators 71, . . . , 7^ of T, we use the injective map 

Hom(r, G) ^ G x • • • x G, y? i-> (^(71), • • • , <p(lk)) 
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to endow Hom(T, G) with the induced topology from the direct product 
G x • • • x G. This topology is independent of the choice of generators. Now, 
suppose G acts continuously on a manifold X. We recall from [6] the following 
definition: 

(1.1) 

R(T,G; X) := {(p G Hom(r, G) : p is injective, and 

<p(T) acts properly discontinuously and freely on X}. 

We remark that our notation here is slightly different from that in [6, 9]: for 
a homogeneous space X = G/H our notation R(T,G;G/ H) here coincides 
with R(T, G, H) loc. cit. 

For each p G R(T,G;X), the quotient space (p(T)\X ~ <p(T)\G/H 
(a Clifford-Klein form of X) becomes a Hausdorff topological space, and 
can be given a unique manifold structure for which the natural quotient 
map X — > (p(T)\X is a local diffeomorphism. Therefore, the Clifford-Klein 
form (p(T)\X enjoys all G-invariant local geometric structures on X. Thus, 
R(T, G; X) may be regarded as a parameter space of Clifford-Klein forms 
ip(T)\X with parameter <p. 

To be more precise on 'parameter', we should take into account 'unessen- 
tial' deformation arising from inner automorphisms. If two homomorphisms 
ip i and </?2 belonging to R(T, G; X) satisfy p 2 = 9 ° <Pi ° for some g G G, 
then the corresponding Clifford-Klein forms are isomorphic to each other by 
the natural diffeomorphism <pi(T)\X ip 2 (r)\X, ipi(T)xH \— > {p 2 (T)gxH. 
In light of this observation, we define the deformation space as the quotient 
set 

(1.2) T(T,G;X):=R(T,G;X)/G. 

For example, if G = PSL(2, M) and X is the upper half plane, and if T 
is the fundamental group of a closed Riemann surface M g of genus g > 2, 
then T(T, G; X) is nothing other than the Teichmuller space of M g . We refer 
the reader to an expository paper [9] for some elementary examples of the 
deformation space for non-Riemannian X. 

Suppose now that <p : T — > G belongs to R(T, G; X), and we reformalize 
(R)' and (S)' as follows: 

(R): (Local rigidity) G • ip is open in Hom(r, G). 
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(S): (Stability) There is an open subset V of Hom(T, G) such that (p E V C 
R(F,G;X). 

We say ip G R(T, G; X) is locally rigid as a discontinuous group for X 
if (R) holds. For a Riemannian symmetric space X, our terminology here is 
consistent with Weil's terminology used in [16]. 

A celebrated Selberg-Weil rigidity [16] for an irreducible Riemannian 
symmetric space X asserts that (R) holds for any torsion free uniform lattice 
(fo(T) of G unless G is locally isomorphic to SL(2,R), whereas R(T, G; X) 
is always open in Hom(T, G) and thus (S) holds. In contrast, there is an 
example that (R) fails for an irreducible non-Riemannian symmetric space 
X of an arbitrarily high dimension (see [6]). 

The failure of (R) arouses our interest in the deformation space T(T, G; X) 
like the classical Teichmuller theory of Riemann surfaces. Besides, the con- 
cept of the stability (S) may be regarded as a first step to understand the 
local structure of the deformation space T(T, G; X) in the setting where (R) 
fails, in particular, where H is non-compact. 

Such a viewpoint traces back to the paper [4] on three dimensional Lorentz 
space forms, where Goldman discovered a discontinuous group V for which 
(R) fails, and raised a question if (S) still holds (not exactly in the way for- 
mulated here). His case concerns with a semisimple Lie group G which is 
locally isomorphic to 5*0(2, 1) x 5*0(2, 1). This question was solved affirma- 
tively, namely, there exists a cocompact discontinuous group V for which (R) 
fails but (S) holds in the generality that G is a semisimple Lie group which is 
locally isomorphic to the direct product of two copies of 50 (n, 1) or SU (n, 1) 
(see [8, 15]). Its proof relies on the criterion for properly discontinuous ac- 
tions [3, 7] on homogeneous spaces of reductive groups. 

For a more general (r, G, X) such as the affine transformation group G, 
both (R) and (S) can fail, as is seen by the following one dimensional example: 

Example 1.1. Let T := Z, and G be the ax + b group, that is, G = {(a, b) : 
a > 0, b G M} with the multiplication given by (a, b) • (a', b') = (aa', ab' + b). 
Consider the affine transformation of G on X : = KL Then, Hom(r, G) ~ G, 
whereas R(T, G; X) ~ {(1,6) : b ^ 0}. Hence, neither (R) nor (S) holds. 

The above example deals with a homogeneous space X of a solvable group 
G and with a cocompact discontinuous group Y. 
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1.2 Summary of this article 

This article analyses the failure of (R) and (S) for homogeneous spaces of 
nilpotent Lie groups G. For a simply-connected nilpotent Lie group G, 
R(T, G; X) becomes open in Hom(T, G) and thus (S) always holds if T\X 
is compact ([19]). Thus, our interest here is in the case when T\X is non- 
compact. As a test case, we initiate a detailed analysis on the deformation 
space in the following setting: 

T := Z fe (free abelian group of rank k), 

X:=R k+1 (nilmanifold), 
G C Aff(R fe+1 ) (a two-step nilpotent subgroup), 

where Y acts on X as nilpotent affine transformations via G. Then, we 
propose a method of giving a concrete description of R(T, G; X) and the 
deformation space T(T, G; X) for a specific choice of G. Our main results 
are Theorems 2.3 and 5.1. 

Besides, we shall see in Corollary 5.1.1 that the deformation space con- 
tains a smooth manifold T'(T, G; X) as its open dense subset such that 

{2k 2 - 1 (k : even), 
- 2 (k : odd, > 3), 
2 (*=1). 

Thus, local rigidity (R) fails for any k because the dimension of the deforma- 
tion space is positive. In the above formula, one sees that the dimension of the 
deformation space T(T, G; X) has a different feature according to whether k 
is even or odd. This will be explained by the criterion of properly discon- 
tinuous actions which involves the existence of a non-zero real eigenvalue of 
a certain k x k matrix, whence the parity of k counts. Moreover, it follows 
from the complete description of R(T, G; X) that we can determine for which 
(fio E R(T, G\ X) the stability (S) fails. 

Our specific choice of G was motivated by Lipsman's classification [11] of 
maximal nilpotent affine transformation groups on M 3 , in which the two-step 
nilpotent group G^for k = 2 played a crucial role. It is noteworthy that 
for any subgroup G of the affine transformation group Aff (M fc+1 ) containing 
our specific G, R(T, G; X) is not open in Hom(r, G) by Theorem 2.3, and 
consequently both (R) and (S) fail. 
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The key idea of our proof is to take a connected subgroup L that contains 
T as a cocompact discrete subgroup, and then to show that every injective 
homomorphism from V into G extends uniquely to a continuous homomor- 
phism from L into G (an idea of syndetic hull). A second step is to find 
explicitly Hom(L, G) in place of Hom(r,G'), and to determine which homo- 
morphism yields a properly discontinuous action. Unlike the reductive case 
[3, 5, 7], properly discontinuous actions for afline transformation groups on 
M, k+l are still far from being fully understood in general, as one sees from 
the current status of the long-standing Auslander conjecture (see [1] and 
references therein). However, fortunately in our special setting, we can use 
the criterion [13] of proper actions for two-step nilpotent Lie groups, which 
was obtained as an affirmative solution to Lipsman's conjecture [11]. Then, 
the final step to the proof of Theorem 2.3 (the description of R(T, G; X)) is 
reduced to a certain problem of Lie algebras, which we can solve explicitly. 



2 Description of deformation parameter 

This section gives a complete description of the parameter space i?(Z fc , G; R k+1 ) 
of properly discontinuous Z fc -actions on IR fc+1 through a certain nilpotent 
afline transformation group G. This is the first of the main results of this 
paper, and is stated in Theorem 2.3. Building on it, we shall determine 
the deformation space T(T, G; R k+1 ) ~ R(T,G;R k+1 )/G in Section 5 (see 
Theorem 5.1). 



2.1 Nilpotent affine transformation group 

We fix a positive integer k. Our basic setting in this paper is: 



(2.1) 



r := 



G :-- 



H := 
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Then G is a simply connected two-step nilpotent Lie group, and the homoge- 
neous space G/H is diffeomorphic to R fe+1 . We shall first give a description of 
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Hom(r, G) in Proposition 2.2.1, and then determine explicitly R(T, G; R fc+1 ) 
as its subset in Theorem 2.3. 

Geometrically, this means that we determine all possible properly discon- 
tinuous affine actions of Z fe on R fe+1 preserving differential forms d£, k+ i and 
d£i A d^k+i (1 < i < k), where . . . , £fc+i) is the coordinate of R fc+1 . 

2.2 Description of Hom(T, G) 

Any group homomorphism ip : T — > G is determined by its evaluation at 
generators of T. Taking a standard basis {ei, . . . , e k } of the abelian group 
T = Z fc , we regard Hom(r, G) as a subset of the direct product G x • • • x G 
by the evaluation map: 

(2.2) Hom(r, G) G x • • • x G, p ^ (^(ei), . . . , <p(e k )). 

Let us describe the image of the injective map (2.2). For this, first we set 

(2.3) Mi := {(x,Y,z) G R fc © M(fc, R) © R fc : 0} C M(k, k + 2; R), 

(2.4) M 2 := M(fc,2fc;R). 

Then, dimMi = fc(fc + 2) and dimM 2 = 2k 2 . Second, for x , y E R fe and 
z G R, we define a (A; + 2) x (k + 2) matrix by 

(0* x j/\ /l k x y + ^zx\ 
z = 1 z 
0/ \0 1 / 

With this expression, our groups G and H (see Section 2.1) are expressed 

as 

G = {g(x, y,z) e R fc , z e R}, 

if = 0,0) : x G R fc }. 

Now we define maps 

: Mi — ► G x • • • x G (i = 1, 2) 
such that their jth (1 < j < k) components are respectively given by 
(2-6) 

Y, z)j := #(^x, yj, zj) for V = (y 1: ■ ■ • , y k ), *z = (zi, • • ■ , z fc ), 
(2.7) * 2 (X,y),- :=#g,g,0) for X = (£, • • • , x k ), Y = (y u ■ ■ ■ , y k ). 
Then, the topological space Hom(r, G) is described via (2.2) as follows: 



7 



Proposition 2.2.1 (Description of Hom(T, G)). 

1) The maps and induce a bijection 

$!U$ 2 :MiUM 2 A Hom(r, G). 

In particular, and ^2 are injective, and their images ^i(Mi) and ^2(^2) 
are disjoint subsets contained in B.om(T,G). 

2) (closure relation) ^/ 2 (M 2 ) is closed in Hom(r,G') 7 whereas the closure of 
^i(Mi) is given as 

(2.8) ^i(Mi) = *i(Mi) U ^2(M 2 d ). 
Here, M$ is a subset of M 2 defined by 

(2.9) M 2 d := {(X, Y) : X, Y G M(k, R), rankX < 1 }. 
We shall give a proof of this proposition in Section 3. 

2.3 Description of R(V, G\ M fc+1 ) 

Let us introduce the following subsets of M x and M 2 : 

(2.10) M{ := {(x, Y, 1) G M(k, k + 2; R) : 2 ^ , rank(*F, z ) = fc } 

(2.11) M 2 r := {(X, Y~) G M(fc, 2k; R) : det(Y - AX) ^ for any A G R }. 

We are now ready to characterize R(T, G; R k+1 ) as a subset of Hom(r, G). 
Here is the first of the main results in this paper: 

Theorem 2.3 (Description of R(T, G; R k+1 )). Let G be a nilpotent Lie 
group defined as (2.1) and T = Z h . Then, the maps ^1 and ^/ 2 (see (2.6) 
and (2.7)) induce the bijection 

f!Uf 2 : M{ U Ml R(F, G; R k+1 ). 

We shall give a proof of this theorem in Section 4. 

2.4 Generic points of R(T, G\ R k+1 ) 

This subsection studies a generic part of R(T, G; R k+1 ) by analysing the sets 
M[ and M 2 r in detail. 
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For X, Y G M(k, R), we define a polynomial of A by 

k 

(2.12) /(X, F; A) := det(Y - AX) = ^ a,(X, Y)X l . 

1=0 

Here, we note 

a (X,Y) = detY, 

k 

a fc _!(X,y) = ^(-ly+^detx^, 

a fe (X,Y) = (-l) fe detX, 

where X^ denotes the submatrix obtained by deleting row i and column j 
from X. 

Let us recall from (2.11) and (2.12) that 

M 2 r = {(X, Y) G M(k, 2k; R) : f(X, Y; A) ^ for any A G R }. 

In order to give a 'generic' part of -R(r, G; R fe+1 ) by means of Theorem 2.3, 
we set 



M r 2 n {(X, Y) : det X ^ 0} (k: even), 

Ml n {(X, Y) : rankX = Jfe - 1, a fc _i(X, y) ^ 0} (Jfe: odd). 



Proposition 2.4.1. 

1) M[ zs open dense in M\. In particular, it has dimension k(k + 2). 

2) M|° is open in Ml, and the complement of M|° m Ml has a smaller 
dimension than that of M™. The dimension of M™ is given by 

,. , ,m 1 (&.' even), 

dimM 9 ro = <^ n v ; ' 

2 [2A; 2 - 1 (k: odd). 

Proof. 1) Clear. 

2) For an even integer /, we consider a monic polynomial of the real 
variable x: 

g (x) = x l + hi^x 1 ' 1 H h hx + b . 

Then, it attains its minimum, denoted by m(bo, b±, ■ ■ ■ , fy_i), which is a con- 
tinuous function of the real coefficients b , b±, ■ ■ ■ , 
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Case 1 (k : even) Suppose (X, F) G M|°. Then, the monic polynomial 
f ^'*v} must be positive for all A G R. Thus we have 

a k (X,Y) r 

M? = ((X,F) G M 2 : detX ^ 0, m ' ' > ^=t££t) >° 



a fc (X,F)' ' a fc (X,F) 
Hence, M 2 ro is open in M 2 = M(fc, 2k; R). 

To see M 2 ro ^ 0, we set J k := I q 2 J G M(Jfe,R). Then (J fc , 4) G 



M|° because 



1 A 



/(J fc ,J fc ;A)= (det(J A =(1 + A 2 )^>0. 

Hence, M\° ^ and dimM 2 ro = dimM 2 = 2k 2 . 

Next, suppose (X, Y) G M 2 r \ M 2 ro . Then detX = by definition. Fur- 
thermore, it follows from f(X,Y; A) ^ for any A G R that the coefficient 

a k -i(X,Y) of A fc_1 must vanish because f(X,Y; A) = a k -i(X, F)A fc_1 H h 

a (X, F) and A: — 1 is odd. Thus we have seen that 

M 2 r \ M 2 ro C {(X, F) G M 2 : det X = a fe _i(X, F) = }. 

Hence the complement M 2 \ M|° has at least codimension two in M 2 . 
Case 2 (A;: odd) First, we claim 

M 2 r C {(X,F) G M(A;,2A;;R) : detX = 0}. 

In fact, since k is odd, the polynomial /(X, F; A) of the real variable A 
has zeros unless the top term a k (X, Y)\ k vanishes. Therefore, a k (X,Y) (= 
(-l) fc detX) = if (X,F) G M 2 r . 

Next, let us prove that M|° is open in the set 

S : = {(X,F) G M(A;,2A;;R) : detX = 0, graddetX ^ 0} 
= {(X, F) G M(k, 2k; R) : rankX = k - 1 }. 

Suppose (X, F) G M (fc, 2*;; R) satisfies 

detX = and a k -i(X,Y) ^ 0. 
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Then f(X, Y; A) 7^ for any A G R if and only if the monic polynomial 



f(X,Y;\) = k _, ^ ai (X,Y) 
a k -i(X,Y) f^a k ^(X,Y) 

is positive for all A. Thus, we have seen 

f rankX = k - 1, a k _±(X,Y) ^ 0, 

M? = I (X, Y) G M(k, 2k; R) : / ao(x , y) a fc , 2 (x,y) \ 

It is now clear that M™ is open in S. To see M™ 7^ 0, we set J' k := 
'•//, , \ 

G M(k; R). Then, (4,4) G M 2 ro because 

o / 

/(4,4;A) = (1 + A 2 )V> . 

Hence, M|° is a non-empty open subset of S. Since S is a non-singular 
manifold of dimension 2k 2 — 1, so is M|°. 

Finally, it follows from the definition of M™ that M£ \ M™ is contained 
in the algebraic variety: 

{(X,Y) : detX = 0, a k _i(X, Y)\\ graddetX|| 2 = 0}, 

which is of dimension 2k 2 — 2. Thus, Proposition 2.4.1 has been proved. □ 

Remark 2.4.2. Proposition 2.4.1 implies that M£ contains an open subset 
of M 2 if and only if k is even. However, M£ itself is not open in M 2 even if 
k is even because (0,4) G M£ is not an inner point, as we shall see in the 
following example: 

Example 2.4.3. Take a half dimensional afnne subspace 

V = {(X,I k ):XeM(k,R)} 

of M(k, 2k; R). Then we have 

V n M 2 r = {(X, I k ) : any eigenvalue of X is in C \ R x }. 

This gives a partial information on R(T, G; R fc+1 ), and was proved in Lipsman 
[11, Theorem 4.4] for k = 2 as a crucial step to the classification of maximal 
nilpotent affine subgroups that act properly on R 3 , and was generalized in 
[12] for k > 3. Our proof here is different and simpler. 
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3 Description of Hom(T, G) 



This section determines Hom(r, G) explicitly, and gives a proof of Proposi- 
tion 2.2.1. 

3.1 Parametrization of Hom(T, G) 

Recall from (2.2) that any p G Hom(T, G) is determined by tp(ej) (1 < 
j < k), which we write as p{ej) — 9{ x j,Vj, z j) f° r some Xj, i/j G R fc and 
Zj G R according to (2.5). Collecting these data Xj,yj (1 < j < k) and 
z := \z±, . . . ,Zk), we obtain an injective map defined by 

(3.1) Hom(r, G) -> M (fc, 2A; + 1; R), <p h-> . . . , ^ fe ; y u . . . , y k ; 1) . 

Let us determine the image of (3.1). Since cp G Hom(r, G) satisfies 

tp(ei)<p{ej) = (p{ei + ej) = (p(ej)(p(ei) 
for any i,j (1 < i,j < k), we have 

(3.2) ZiXj = ZjXi for any i,j (1 <i,j < k). 

Conversely, given k elements gi, • ■ • ,g k in G that mutually commute, we can 
define a group homomorphism tp : Y — > G by ^(S^i m j e j) := fi'r 1 ' ' 'fiC*- 
Therefore, the image of (3.1) is characterized by the condition (3.2), that is, 
we have a bijection: 

Hom(r, G) ~ { ^i, . . . , a^; F; 2*j : F G M(fc, R); xl, . . . , x k , ~z satisfies (3.2)}. 

We shall find all solutions of (3.2), according to the following two cases: (a) 
2 ^ and (b) 1=0. 

In the case (a), there exists uniquely an element r G R fc such that Xj = 
ZjT for all j (1 < j < k). This amounts to \E'i(M 1 ) (see (2.6) for the definition 
of 

In the case (b), any Xi, . . . , x k solves (3.2). This amounts to ^2(^2) (see 
(2.7) for the definition of * 2 ). 

Hence, Hom(r, G) is the disjoint union of ^i(Mi) and # 2 (M 2 ). Thus we 
have completed the proof of Proposition 2.2.1 (1). 

□ 
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3.2 Closure relation in Hom(T, G) 

This subsection gives a proof of Proposition 2.2.1 (2). It is clear that \I/ 2 (M 2 ) 
is a closed set. Let us consider the closure of \&i(Mi), and find its boundary. 
What we need is to prove: 

*i(Mi) n # 2 (M 2 ) = ^ 2 (M 2 d ). 

Proof of the inclusion D : Suppose (X, Y) G # 2 (M|). Since rankX < 1, 
we find x G R k and a G IR fc \ {0} such that X = 'x f a. In light of the obvious 
formula 

g(a j x,y j ,0) = lim g(-j-lx, yj, (1 < j < fc), 
we conclude from the definitions (2.6) and (2.7) of \&i and ^ 2 that 

* 2 (x,y) = iim* 1 (zx,y,y). 

i— >oo i 

As (lx,Y, — ) (/ = 1,2, •••) is a sequence of M 1? we have proved the 
inclusion D. 

Proof of the inclusion C : Take any sequence (x^ l \ Y^ l \ z®) in Mi such 

that ^i(x^ l \Y^,z^) converges to an element of # 2 (M 2 ), say, V 2 (X,Y) for 
some X, y G M(fc, R). Then the formula 

lim Vi(x m ,Y( l \z m ) = V 2 {X,Y) 
i— +00 

implies that X is the limit of X® := (z^x®,--- ,zf ^) as / tends to 
infinity. Since rankX^ < 1, its limit also satisfies rankX < 1. Thus we 
have proved the inclusion C. 

Thus, Proposition 2.2.1 (2) is proved. □ 



4 Proof of Theorem 2.3 

This section gives a proof of Theorem 2.3. Our strategy here is to rewrite 
the condition of R(T, G; R fc+1 ), in particular, the condition for properly dis- 
continuous actions in the following scheme: 

T discrete subgroup (see (1.1)) 

=>- L = T its syndetic hull (see Proposition 4.3.2) 
=>- [ its Lie algebra. (see Section 4.4) 
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4.1 Proper actions and properly discontinuous actions 



In dealing with properly discontinuous actions of a discrete group, a more 
general notion "proper action" is sometimes useful. We recall: 

Definition 4.1.1 (Palais [14]). Suppose that a locally compact topological 
group L acts continuously on a Hausdorff, locally compact space X. For a 
subset S of X, we define a subset of L by L s = {7 G L : fl S 7^ 0}. The 
L-action on X is said to be proper if Ls is compact for every compact subset 
SoiX. 

We note that the L-action is properly discontinuous if L is a discrete 
group and if the L-action is proper. 

The following elementary observation is a bridge between the action of a 
discrete group and that of a connected group. 

Observation 4.1.2 ([5, Lemma 2.3]). Suppose a locally compact group L 
acts on a Hausdorff, locally compact space X . Let Y be a cocompact discrete 
subgroup of L. Then 

1) The L-action on X is proper if and only if the Y-action is properly 
discontinuous. 

2) L\X is compact if and only if Y\X is compact. 

4.2 Extension from a discrete subgroup 

Suppose we are in the setting of Section 2.1. We set 

L := R k 

and regard r = Z k as a cocompact discrete subgroup of L. We write 
Hom(L, G) for the set of continuous group homomorphisms from L into G. 
In our setting (2.1), every homomorphism from Y into G extends uniquely 
to a continuous homomorphism from L to G. That is, we have: 

Lemma 4.2.1. The restriction map Hom(L, G) — > Hom(T, G), ip 1— > ?/>|r is 
bijective. 
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Proof. As G is a simply connected nilpotent group, the exponential map, 
exp : q — > G is bijective. We write log for its inverse. Then, if) e Hom(L, G) 
satisfies 

(4.1) ^(^^ °J e j) = ex P(^Z 0j ^°S^( e J')) ^ or an y Gl ' ' ' ' ' flfc e ^ 

3=1 3=1 

This shows that the homomorphism if> is determined by its restriction if)\r- 
Conversely, the formula (4.1) also indicates how to extend a homomorphism 
from T to L. Thus we have proved Lemma 4.2.1. □ 

In light of Lemma 4.2.1, any property of if) should be expressed in terms 
of the restriction if)\r in principle. We show: 

Lemma 4.2.2. The following two conditions on if) e Hom(L, G) are equiva- 
lent: 

(i) if) is injective. 

(ii) if)\r is injective and if)(T) is discrete in G. 

Proof. Since G is a simply connected nilpotent Lie group, any connected 
subgroup of G is closed. Therefore, if> : L/Kerif> — > G is a homeomorphism 
onto a closed subgroup of G. In particular, if>(T) is discrete in G if and only 
if T/r n Ker?/> is discrete in Lj Kerip. Now, it is clear that (i) implies (ii). 

Conversely, if if> is not injective and if if)\r is injective, then the compo- 
sition map rcL-> L/Kerif) is injective with non-discrete image because 
rankT < dim(L/ Ker if)). Hence, if)(T) is not discrete in G, too. Thus, (ii) 
also implies (i). □ 

4.3 A continuous analogue of properly discontinuous 
actions 

Following Observation 4.1.2, we amplify Lemma 4.2.2 with the condition of 
proper actions on the homogeneous space G/H: 

Lemma 4.3.1. Let if) G Hom(L, G) and ip = if)\ r (see Lemma 4-2.1). Then 
the following two conditions are equivalent: 

(i) if) : L — > G is injective and if>{L) acts properly on G/H. 
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(ii) ip : T — * G is injective andipiT) acts properly dis continuously and freely 
on G/H. 



Proof, (i) (ii): Since ip is injective, it follows from Lemma 4.2.2 that (f(T) 
is discrete in a closed subgroup ip(L). Therefore, <p(T) acts properly discon- 
tinuously on G/H because ip(L) acts properly on G/H. Furthermore, any 
properly discontinuous action of <p(T) is automatically free because (p(T) ~ T 
is torsion-free. Hence (ii) is proved. 

(ii) =>- (i): If y?(r) acts properly discontinuously on G/H then (p(T) is 
discrete in G. Hence, ip : L — > G is injective by Lemma 4.2.2. Further- 
more, ip(L) with its relative topology contains (p(T) as a cocompact discrete 
subgroup. Therefore, ip(L) acts properly on G/H by Observation 4.1.2 (1). 
Thus, we have proved the implication (ii) =>- (i). □ 

We are ready to characterize R(T, G; R fc+1 ) by means of the connected 
subgroup L: 

Proposition 4.3.2. Under the isomorphism Hom(L, G) —> Hom(r, G) in 
Lemma 4-2.1, we have 

R(T, G; R k+1 ) ~ G Hom(L, G) : i) ip is injective, 

ii) V ; (-^) ac t s properly on G/H}. 



So far, we have transferred proper discontinuity and freeness of discrete group 
actions into a certain property of connected group actions. Now, let us 
rewrite the latter condition in terms of Lie algebras. We use the German 
lower case letters q, f) and I to denote the Lie algebras of G, H and L 
respectively. We write dip for the differential of ip e Hom(L, G). Consider 
the following conditions on dip: 

(4.2) dip : [ — > q is injective, 



4.4 Reformulation of R(T, G; R k+1 ) 



(4.3) 




g&G 



Now we can restate Proposition 4.3.2 as 
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Proposition 4.4.1. Under the isomorphism Hom(L, G) Hom(r, G) (see 
Lemma \.2.1), we have 

R(T, G; R k+1 ) ~ {4) e Hom(L, G) : dip satisfies (4.2) and (4.3) }. 



Proof. Any connected subgroup of a simply connected nilpotent Lie group is 
simply connected. Hence, dtp is injective if and only if tp is injective. Now 
use the criterion of proper actions for a homogeneous space of a two-step 



Lemma 4.4.2 ([13, Theorem 2.11]). Let G be a simply connected Lie 
group, and H, L its closed subgroups. Suppose G is a two-step nilpotent Lie 
group, which means that the commutator subgroup of G is contained in the 
centre of G. Then, the following three conditions on ip are equivalent. 

(i) ip(L) acts on G/H properly. 



Remark 4.4.3. Lemma 4.4.2 gives an affirmative solution to Lipsman's con- 
jecture [11] for two-step nilpotent Lie groups. Recently, Baklouti-Khlif [2] 
and Yoshino [18] proved independently that Lipsman's conjecture is still true 
for three-step nilpotent Lie groups. 

4.5 Completion of the proof of Theorem 2.3 

Now let us complete the proof of Theorem 2.3. We have already reduced it 
to a problem of Lie algebras. Now, we use the following: 

Lemma 4.5.1. We define the variety in q by 



nilpotent Lie group G as follows. 



□ 



(ii) ^(LjngHg- 1 = {e} for all g G G. 

(iii) #([)nM eG Ad(^)[) = {0}. 



V= |J Ad(g)t). 



Then we have 



(4.4) 
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Proof. Elementary computation. 



□ 



According to the parametrization 



$ 1 U* 2 :M 1 UM 2 ^ Hom(L, G) ^ Hom(r, G) 



given in Proposition 2.2.1 and Lemma 4.2.1, we examine if dip satisfies (4.2) 
and (4.3) for ip G ^i(Mi) and ip G ^/ 2 (M 2 ), respectively. The following 
proposition gives criteria for (4.2) and (4.3): 

Proposition 4.5.2. 

1) Let ip := ^/i(x,Y, z) for (x,Y, z) G Mi. Then, we have the following 
equivalence: 




In this case, ip satisfies (4.3), too. 

2) Let i) := V 2 (X, Y) for (X, Y) G M 2 . Then 




In this case, we have the following equivalence: 



ip satisfies (4.3) 



det(y - bX) ^ /or any 6 G R. 



Proof. 1) It follows from the definition of (see (2.6)) that 



(4.5) 




(a, z)x Y a 
(a, ~z 





Here, ( , ) denotes the standard inner product on R k . Then 



■0 satisfies (4.2) 



{a el : (z, a) = 0, Ya = 0} = {0} 
rank(*Y, ~z) = k. 



Furthermore, by (4.4) and (4.5) 



dip(a) G V 



dip(a) = 0. 
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Therefore, (4.2) implies (4.3). 

2) It follows from the definition of ^ 2 (see (2.7)) that 

'0 Xa Ya' 
(4.6) #(a) = I 







Then, 



i> satisfies (4.2) {ael: Xa = 0, Ya = } = {0} 
~ rank (*) = k. 

Suppose (4.2) is satisfied. It follows from (4.4) and (4.6) that 
ip satisfies (4.3) 

there is no b G R and a G R fc such that Ya = bXa ^ 
•<=>- det(Y — bX) = has no real solution for any b G R. 

Hence, Proposition is proved. □ 
We note that if det(Y - bX) ^ for any b G R then rank ( J = k 



Y / 

because det Y ^ 0. Then, Theorem 2.3 follows from Propositions 4.4.1 and 
4.5.2. Hence, we have completed the proof of Theorem 2.3. □ 



5 Deformation space 

Building on the description of the parameter space R(T, G; R fc+1 ) of discon- 
tinuous groups given in Theorem 2.3, we determine explicitly the deformation 
space T(r, G; R fe+1 ). This is stated in Theorem 5.1 and Corollary 5.1.1, and 
is the second of the main results of this paper. 

5.1 Description of the deformation space T(T, G; R k+1 ) 
We define subsets of M[ and M£, respectively, by 
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(5.1) 

Di ■= I (x, ...,m),~z)t M(k, k + 2;\ 



1) z^O, 

2) rfjJLz (l<j<k), 

3) rank(?fi, • • • , ff k ) = k - 1. 



(5.2) 

^■-{(X,Y)eM(k,R)^M(k,R): % f^y^xJ^O for anyAel. }' 

We note that the third condition in (5.1) asserts that rank(?7i, • • • ,rfk) 
attains its maximum because all the vectors rfj (1 < j < k) are orthogonal 
to z. 

We retain the setting as in Section 2.1. In particular, T = Z k and G is 
a nilpotent affine transformation group defined in (2.1). For i — 1, 2, we 
denote by 

% : MT -> T(r, G; M fc+1 ) 

the composition of ^ : M[ -> i2(r, G; M fe+1 ) (see (2.6) and (2.7)) and the 
natural quotient map R(T, G; R k+1 ) -> T(r, G; M fc+1 ). 

Here is an explicit description of the deformation space: 
Theorem 5.1. The maps and induce the following bijection: 

$iUf 2 :AUD 2 ^ T(T, G; R k+1 ). 

In particular, we find the dimension of the deformation space: 

Corollary 5.1.1. The deformation space T(T, G; R fe+1 ) contains a smooth 
manifold T as its open dense subset, where the dimension of T is given by 

{2k 2 — 1 (k : even), 
2k 2 - 2 (k : odd, > 3), 
2 (* = 1). 
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5.2 Proof of Theorem 5.1 and Corollary 5.1.1 

and M(k,2k;M), respectively, as follows: 



We let G act on M(k, k + 2; 

'h a b 
for h = I 1 c 
1 



G G, the actions of h are given by 



(5.3) (x, Y, ~z) i-> (x; Y + (a - cf fz , ~z) 

(5.4) (X,F)^(X,r-cX) 



on M(k,k + 2; 
on M(jfe,2ife;R). 



Proposition 5.2.1. 

1) 5otfi Mi and M[ are G-stable subsets of M(k, k + 2;R). 

2) M 2 r a G-stable subset of M 2 = M(k,2k;R). 

3) For « = 1, 2, £/ie maps ^ : M? — > Hom(r, G) respect G-actions. 

4) For i = 1, 2, F>j are complete representatives of the G -orbit on M[. 

Proof. 1) Clear from the definitions (2.3) and (2.10) of M x and M[. 

2) Clear from the definition (2.11) of M 2 r . 

3) We first note that via (2.2) the G-action on Hom(r, G) is compatible with 
the diagonal G-action on G x • • • x G: 

(9i,--',9k) ^ (hgxh' 1 ,- ■ ■ ,hg k h~ l ). 

Now, we compute the j'th components of the image of (see (2.6) for the 
definition) and (see (2.7) for the definition), respectively, as follows: 
'h a b _ 

G G, we have 



For h = 




hexp 





ZjX 




\ h 1 = exp 1 







z 


\o 





o 





ZjX yj + Zj(a — cx) s 
Zi 







and 



/iexp 




exp 



[0 


\0 







Xj yj cxj 





This is what we wanted to prove. 

4) This is an elementary linear algebra. 



□ 
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Then, Theorem 5.1 is an immediate consequence of Theorem 2.3 and 
Proposition 5.2.1 (4). Corollary 5.1.1 now follows from Proposition 2.4.1, 
and from the G-action on R(T, G; M. k+1 ) described in the above proof. 
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